QA MODEL

Does Named Entity Recognition Truly Not Scale Up to Real-world Product Attribute Extraction? (2023.emnlp-industry)

Challenge:	Multi-hop question answering (QA) requires an information retrieval system that can find multiple supporting evidence needed to answer the question.
Approach:	They propose a technique that uses information of entities present in the initial retrieved evidence to learn to ‘hop’ onto other relevant evidence.
Outcome:	The proposed method boosts retrieval performance on a multi-hop question answering dataset with 5 million Wikipedia paragraphs and a model without training increases its performance by 10.59 F1.

Copied to clipboard

Wei-Te Chen, Keiji Shinzato, Naoki Yoshinaga, Yandi Xia

Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs (2020.acl-main)

Challenge:	scalability of attribute-value extraction (AVE) task is key for a large number of products . a question-answering (QA)-based approach is better for AVE, but requires a larger number of classes to be scalable.
Approach:	They propose a question-answering-based approach that additionally inputs the target attribute as a query to extract its values.
Outcome:	The proposed approach outperforms a classical approach on real-word e-commerce datasets in accuracy and speed.

Copied to clipboard

Dong Bok Lee, Seanie Lee, Woo Tae Jeong, Donghwan Kim, Sung Ju Hwang

Domain-agnostic Question-Answering with Adversarial Training (D19-58)

Challenge:	Existing models for question answering are limited in the availability of labeled data.
Approach:	They propose a hierarchical conditional variational autoencoder for generating QA pairs given unstructured texts as contexts while maximizing mutual information between generated QA pair to ensure consistency.
Outcome:	The proposed framework achieves impressive performance gains over baseline models on both tasks, using only a fraction of data for training.

Copied to clipboard

Seanie Lee, Donggyu Kim, Jangwon Park

DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries (2022.lrec-1)

Challenge:	Adapting models to new domain without finetuning is a challenging problem in deep learning.
Approach:	They propose an adversarial training framework for domain generalization in Question Answering task using a conventional QA model and a discriminator.
Outcome:	The proposed model outperforms the baseline model on Question Answering (QA) task.

Copied to clipboard

Jayetri Bardhan, Anthony Colas, Kirk Roberts, Daisy Zhe Wang

Multimodality

Challenge:	a new question answering dataset is being developed for electronic health records . structured tables and unstructured notes can be duplicated, contradictory or provide additional context .
Approach:	They develop a question-answer-matching dataset using structured tables and unstructured notes from an EHR.
Outcome:	The proposed model is based on a model with a modality selection network . it uses the prediction of a RAT-SQL to choose between EHR tables and clinical notes .

Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering (2022.findings-naacl)

Copied to clipboard

Jeonghwan Kim, Junmo Kang, Kyung-min Kim, Giwon Hong, Sung-Hyon Myaeng

Learning to Collaborate for Question Answering and Asking (N18-1)

Challenge:	Existing numerical reasoning models overly rely on parametric knowledge at inference time . previous studies show that understanding numbers in text improves numerical reasoning accuracy .
Approach:	They propose a numerical reasoning model that leverages parametric knowledge to alleviate this over-reliance on parametric information.
Outcome:	The proposed model improves numerical reasoning accuracy and performance in DROP.

Copied to clipboard

Duyu Tang, Nan Duan, Zhao Yan, Zhirui Zhang, Yibo Sun, Shujie Liu, Yuanhua Lv, Ming Zhou

QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation (2022.emnlp-main)

Challenge:	Question answering (QA) and question generation (QG) are closely related tasks.
Approach:	They propose a training algorithm that generalizes both Generative Adversarial Network and Generating Domain-Adaptive Nets under the question answering scenario.
Outcome:	The proposed training algorithm generalizes both Generative Adversarial Network (GAN) and Generating Domain-Adaptive Nets (GDAN) under the question answering scenario.

Copied to clipboard

Zhenrui Yue, Huimin Zeng, Bernhard Kratzwald, Stefan Feuerriegel, Dong Wang

Graph Guided Question Answer Generation for Procedural Question-Answering (2024.eacl-long)

Challenge:	Question answering models often suffer from performance deterioration upon deployment .
Approach:	They propose a self-supervised framework called QADA for QA domain adaptation . they propose to augment training QA samples with hidden space augmentation .
Outcome:	The proposed framework improves on multiple target datasets over state-of-the-art methods.

Copied to clipboard

Hai Pham, Isma Hadji, Xinnuo Xu, Ziedune Degutyte, Jay Rainey, Evangelos Kazakos, Afsaneh Fazly, Georgios Tzimiropoulos, Brais Martinez

Efficient and Robust Question Answering from Minimal Context over Documents (P18-1)

Challenge:	a new method for question-answer generation from procedural text is sub-optimal for training QA models.
Approach:	They propose a method for generating exhaustive and high-quality training data from procedural text . they use procedural data to represent each step and the overall flow of the procedure as graphs .
Outcome:	The proposed method outperforms existing methods on task-specific question answering tasks.

Copied to clipboard

Sewon Min, Victor Zhong, Richard Socher, Caiming Xiong

Responsible & Trustworthy NLP

Challenge:	Recent work shows that neural QA models are sensitive to adversarial inputs.
Approach:	They propose a sentence selector to select the minimal set of sentences to feed into a QA model.
Outcome:	The proposed system reduces training time and inference time by up to 13 times . it is comparable to or better than the state-of-the-art on SQuAD, NewsQA, TriviaQA and SQu AD-Open .

UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)

Copied to clipboard

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi

Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation (2022.findings-emnlp)

Challenge:	Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems .
Approach:	They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning .
Outcome:	a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data .

Copied to clipboard

Junru Lu, Xingwei Tan, Gabriele Pergola, Lin Gui, Yulan He

Semantic Text Processing

Challenge:	Existing QA frameworks that use event-centric reasoning are lacking.
Approach:	They propose a novel QA model with contrastive learning and invertible event transformation . they use an invertable transformation matrix to project event vectors into a common event embedding space .
Outcome:	The proposed model achieves 8.4% gain in token-level F1 score and 3.0% gain in Exact Match score on the ESTER dataset.

CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering (2024.emnlp-main)

Copied to clipboard

Yike Wu, Yi Huang, Nan Hu, Yuncheng Hua, Guilin Qi, Jiaoyan Chen, Jeff Pan

Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction (P19-1)

Challenge:	Existing knowledge rewriting methods may include irrelevant information, omit crucial details, or fail to align with the question’s semantics.
Approach:	They propose a new rewriting method CoTKR for generating reasoning traces and corresponding knowledge in an interleaved manner, thereby mitigating the limitations of single-step knowledge rewrite.
Outcome:	The proposed method mitigates the limitations of single-step knowledge rewriting and bridges the preference gap between the knowledge reactor and the question answering (QA) model.

Copied to clipboard

Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, Junji Tomita

Information Extraction & Text Mining

Challenge:	Question answering (QA) using textual sources for purposes such as reading comprehension has attracted much attention.
Approach:	They propose a Query Focused Extractor model for evidence extraction and multi-task learning with the QA model.
Outcome:	The proposed model achieves state-of-the-art evidence extraction score on hotpotQA and FEVER, which is a recognizing textual entailment task on a large textual database.

Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning (2023.acl-long)

Copied to clipboard

Wangzhen Guo, Qinkang Gong, Yanghui Rao, Hanjiang Lai

Finding Generalizable Evidence by Learning to Convince Q&A Models (D19-1)

Challenge:	Existing QA models rely on shortcuts to provide the true answer, referred to as disconnected reasoning problem.
Approach:	They propose a causal-effect approach that exploits true multi-hop reasoning instead of shortcuts.
Outcome:	The proposed method achieves 5.8% higher points of its Supps score on hotpotQA through true multihop reasoning.

Copied to clipboard

Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance (2021.acl-long)

Challenge:	a system that finds the strongest supporting evidence for a given answer is proposed . a study using passage-based question-answering (QA) shows that agents select evidence that generalizes .
Approach:	They propose a system that finds the strongest supporting evidence for a given answer . they use passage-based question-answering (QA) as a testbed to train evidence agents .
Outcome:	The proposed system improves QA in a robust manner by using agent-selected evidence.

Copied to clipboard

Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-Seng Chua

Structured Data in NLP

Challenge:	Existing QA systems focus on unstructured text, structured knowledge base, or semi-structured tables.
Approach:	They propose a large-scale question answering model based on financial reports . numerical reasoning is usually required to infer the answer .
Outcome:	The proposed model achieves 58.0% inF1, an 11.1% increase over the baseline model, but still lags behind the best human model.

Incorporating External Knowledge into Machine Reading for Generative Question Answering (D19-1)

Copied to clipboard

Bin Bi, Chen Wu, Ming Yan, Wei Wang, Jiangnan Xia, Chenliang Li

Uncertainty Guided Global Memory Improves Multi-Hop Question Answering (2023.emnlp-main)

Challenge:	Existing knowledge-aware QA models do not have commonsense and background knowledge to answer nontrivial questions.
Approach:	They propose a new neural model which exploits external knowledge to generate answers in natural language for a given question with context.
Outcome:	The proposed model improves answer quality over existing models without knowledge and knowledge-aware models, a study shows . state officials in Hawaii confirmed that president Barack Obama was born in the U.S.

Copied to clipboard

Alsu Sagirova, Mikhail Burtsev

Relation-Guided Pre-Training for Open-Domain Question Answering (2021.findings-emnlp)

Challenge:	Transformers are used to solve multi-hop question answering tasks that require reasoning over multiple parts of a long document.
Approach:	They propose a method that collects relevant information over the entire document and then combines it with local context to solve a multi-hop question answering task.
Outcome:	The proposed method improves on three MHQA datasets compared to the baseline model.

Copied to clipboard

Ziniu Hu, Yizhou Sun, Kai-Wei Chang

Regularization of Distinct Strategies for Unsupervised Question Generation (2020.findings-emnlp)

Challenge:	Existing QA datasets are imbalanced in some types of relations, which hurts generalization performance over long-tail questions.
Approach:	They propose a relation-guided pre-training framework to infer latent relations from a QA dataset . they then propose RGPT-QA to conduct extractive QA to get the target answer entity .
Outcome:	The proposed framework improves Exact Match accuracy on natural questions, TriviaQA, and WebQuestions.

Copied to clipboard

Junmo Kang, Giwon Hong, Haritz Puerto San Roman, Sung-Hyon Myaeng

Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics (2022.findings-acl)

Challenge:	Unsupervised question answering (UQA) is a task of answering questions from a context that contains the answer.
Approach:	They propose a method to generate higher-quality questions with a teacher-student architecture and a regularization module to avoid bias toward a particular question generation strategy.
Outcome:	The proposed method generates higher-quality questions across diverse datasets and tasks and can be used to create a model with few-shot learning.

Copied to clipboard

Daniel Deutsch, Dan Roth

Summarization

Challenge:	Existing QA-based summarization metrics must automatically determine whether the QA model’s prediction is correct or not.
Approach:	They benchmark lexical answer verification methods used by current QA-based metrics and two more sophisticated text comparison methods, BERTScore and LERC.
Outcome:	The proposed methods outperform the other methods in some settings while remaining statistically indistinguishable from lexical overlap in others.

Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)

Copied to clipboard

Jifan Chen, Eunsol Choi, Greg Durrett

Improving Unsupervised Question Answering via Summarization-Informed Question Generation (2021.emnlp-main)

Challenge:	Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts.
Approach:	They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules.
Outcome:	The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting.

Copied to clipboard

Chenyang Lyu, Lifeng Shang, Yvette Graham, Jennifer Foster, Xin Jiang, Qun Liu

Text Generation

Challenge:	Question Generation (QG) is the production of meaningful questions given a set of input passages and corresponding answers.
Approach:	They propose a method which uses questions generated heuristically from news summaries as a source of training data for a QG system.
Outcome:	The proposed method outperforms previous unsupervised models on three in-domain datasets and three out-of-domain ones.

NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset (2021.findings-emnlp)

Copied to clipboard

Qiyuan Zhang, Lei Wang, Sicheng Yu, Shuohang Wang, Yang Wang, Jing Jiang, Ee-Peng Lim

Long-Tailed Question Answering in an Open World (2023.acl-long)

Challenge:	Existing question answering datasets lack numerical reasoning and reasoning processes . current research on numerical reasoning focuses on simple calculations .
Approach:	They propose a conversational and bilingual question answering dataset with numerical reasoning with compound mathematical expressions.
Outcome:	The proposed model achieves 55.5 exact match scores while human performance is 89.7.

Copied to clipboard

Yi Dai, Hao Lang, Yinhe Zheng, Fei Huang, Yongbin Li

Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering (2020.acl-main)

Challenge:	Existing QA approaches require access to seen tasks or do not explicitly model samples from unseen tasks.
Approach:	They propose an open-tailed QA model that encourages knowledge sharing between head, tail and unseen tasks and explicitly mines knowledge from a large pre-trained language model.
Outcome:	The proposed model outperforms the state-of-the-art on a large-scale dataset.

Copied to clipboard

Alexander Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, Bing Xiang

Understanding Unnatural Questions Improves Reasoning over Text (2020.coling-main)

Challenge:	Question Answering (QA) is a field of increasing demand due to the availability of information online.
Approach:	They propose an unsupervised approach to training QA models with generated pseudo-training data by applying a simple template on a related sentence rather than the original context sentence.
Outcome:	The proposed approach improves the performance of a QA model on generated pseudo-training data.

Copied to clipboard

Xiaoyu Guo, Yuan-Fang Li, Gholamreza Haffari

Responsible & Trustworthy NLP

Challenge:	Complex question answering (CQA) requires large amounts of human-annotated data . learning effective CQA requires large amount of human annotated .
Approach:	They propose to map human-generated questions into unnatural machine-generated ones . they generate synthetic pairs and train a parser that associates synthetic questions with their corresponding action sequences.
Outcome:	The proposed model outperforms the state-of-the-art model trained on human-labeled data.

MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data (2022.acl-long)

Copied to clipboard

Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang

FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020.acl-main)

Challenge:	Existing benchmarks for numerical reasoning over hybrid data only include a single flat table in each document .
Approach:	They propose a new benchmark with QA pairs over multi hierarchical tabular and textual data.
Outcome:	The proposed model is more complex and challenging than existing benchmarks and is available on github . it uses facts retrieving to extract relevant facts from both tables and text and symbolic reasoning over retrieved facts.

Copied to clipboard

Esin Durmus, He He, Mona Diab

Summarization

Challenge:	Existing automatic metrics do not capture errors in abstractive summarization models.
Approach:	They propose an automatic question answering metric for faithfulness that leverages recent advances in reading comprehension.
Outcome:	The proposed metric has significantly higher correlation with human faithfulness scores on highly abstracted summaries.

Answer Quality Aware Aggregation for Extractive QA Crowdsourcing (2022.findings-emnlp)

Copied to clipboard

Peide Zhu, Zhen Wang, Claudia Hauff, Jie Yang, Avishek Anand

Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering (2021.acl-long)

Challenge:	Existing methods for creating extractive question answering datasets are crowdsourcing, but results are often inconsistent.
Approach:	They propose a method for aggregating answers from different crowd workers that takes into account the relations between the answer, question, and context passage.
Outcome:	The proposed method outperforms baselines by 16% on precision and effectively conduct answer aggregation for extractive question answering task.

Copied to clipboard

Gangwoo Kim, Hyunjae Kim, Jungsoo Park, Jaewoo Kang

Selective Question Answering under Domain Shift (2020.acl-main)

Challenge:	Existing approaches do not explicitly train QA models on how to resolve conversational dependency, and thus these models are limited in understanding human dialogues.
Approach:	They propose a framework that generates self-contained questions that can be understood without the conversation history and then trains a QA model with the pairs of original and self-constructed questions using a consistency-based regularizer.
Outcome:	The proposed framework improves the models’ performance by up to 1.2 F1 on QuAC, and 5.2 F1 for CANARD, while addressing the limitations of the existing approaches.

Copied to clipboard

Amita Kamath, Robin Jia, Percy Liang

Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (D19-1)

Challenge:	Deployed question answering (QA) models need to know when to abstain from answering questions that diverge from their training data.
Approach:	They propose a selective question answering under domain shift in which a QA model is tested on a mixture of in-domain and out-of-domain data and must answer (i.e., not abstain on) as many questions as possible.
Outcome:	The proposed method answers 56% of questions while maintaining 80% accuracy.

Copied to clipboard

Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, Bing Xiang

Harvesting and Refining Question-Answer Pairs for Unsupervised QA (2020.acl-main)

Challenge:	Existing studies have shown that BERT models can find answers from multiple passages . however, the results of these studies are still unaddressed.
Approach:	They propose a multi-passage BERT model to globally normalize answer scores across all passages of the same question.
Outcome:	The proposed model outperforms state-of-the-art models on four benchmarks.

Copied to clipboard

Zhongli Li, Wenhui Wang, Li Dong, Furu Wei, Ke Xu

Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement (2022.emnlp-main)

Challenge:	Recent research attempts to extend unsupervised question answering to settings with few or no labeled data available.
Approach:	They propose two approaches to improve unsupervised question answering . first, they harvest lexically and syntactically divergent Wikipedia questions to automatically construct a corpus of question-answer pairs . second, they take advantage of the QA model to extract more appropriate answers .
Outcome:	The proposed approach outperforms previous unsupervised approaches by a large margin and is competitive with early supervised models.

Copied to clipboard

Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Clark

VIMQA: A Vietnamese Dataset for Advanced Reasoning and Explainable Multi-hop Question Answering (2022.lrec-1)

Challenge:	Using simulated feedback, our system (called TeachMe) continually improves with time, and without model retraining.
Approach:	They propose to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections toerroneous model beliefs that users identify during interaction.
Outcome:	The proposed system improves with time and without model retraining, and with real users, by 15% on a hidden test set after teaching.

Copied to clipboard

Khang Le, Hien Nguyen, Tung Le Thanh, Minh Nguyen

Responsible & Trustworthy NLP

Challenge:	Existing Vietnamese Question Answering (QA) datasets do not explore the model’s ability to perform advanced reasoning and provide evidence to explain the answer.
Approach:	They propose to use Vietnamese as a question-answer dataset with 10,000 Wikipedia-based multi-hop question-and-answ pairs to test model's ability to reason and explain the answer.
Outcome:	The proposed dataset is in Vietnamese, a low-resource language.

Unsupervised Question Decomposition for Question Answering (2020.emnlp-main)

Copied to clipboard

Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, Douwe Kiela